Traditional signature analysis may not always able to detect malicious files, especially polymorphic viruses, altered versions of malicious files, and shellcodes. Therefore, modern antivirus applications additionally perform a scan using a so-called “sandbox”—a computer environment for safe execution of processes. The sandbox can be realized, for example, in the form of a virtual machine, on the basis of a partial virtualization of the file system and the registry, on the basis of rules of access to the file system and registry, or on the basis of a hybrid approach. The file being scanned is executed in the sandbox. Events which occur as a result of the execution of the process launched from the file may be saved in a log by means of an intercepting of the various procedures being executed by both the process and the operating system (OS). The antivirus application may then analyze the resulting log. The log usually saves the API (application programming interface) function calls which have been made by the mentioned process during the execution, and also the returns from the called API functions (transfer of control at the return address). The execution of a file in a sandbox usually occurs within a limited period of time (up to several dozen seconds), since the execution of the file in the sandbox and the intercepting of the API function calls by the antivirus application substantially slows down the speed of execution of the file. At the same time, when a file containing a shellcode is executed in a sandbox, its detection by analysis of the API function call logs may be difficult. This is because the shellcode may be loaded into the memory of the process, but the execution of the process in the sandbox was terminated earlier than when control should have been transferred to the memory section containing the shellcode. In this case, a need arises to preserve and analyze the memory dumps.
Analysis of the prior art leads to the conclusion that it may be ineffective and in some instances impossible to employ the previous technologies, whose drawbacks are solved by the present invention, namely, a system and method for detection of malicious code in a file.